Systems and Methods for Image-Based Location Determination and Parking Monitoring

ABSTRACT

Embodiments relate to systems, methods and computer readable media for parking monitoring in an urban area by image processing operations. Embodiments perform parking monitoring by capturing images of an urban area, comparing captured images with reference images to determine location and parking conditions. Embodiments processes captured images to detect licence plates, vehicles or parking signs to determine compliance of vehicles with parking conditions.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 17/910,693, filed on Sep. 9, 2022, which is anational stage application of PCT Application No. PCT/AU2021/050160,filed on Feb. 25, 2021, which claims priority to Australian PatentApplication No. 2020900736, filed on Mar. 10, 2020, and to AustralianPatent Application No. 2020902942, filed on Aug. 18, 2020. The entirecontents of the named applications are incorporated herein by reference.

TECHNICAL FIELD

Embodiments relate to systems and methods for location estimation ordetermination. Embodiments relate to systems and methods for image-basedlocation estimation or determination. Embodiments also relate to systemsand methods of using image-based location estimation or determinationfor parking monitoring.

BACKGROUND

Global Positioning System (GPS) technologies assist in determination oflocation by communication with GPS specialised satellites incommunication with a GPS receiver. Each GPS satellite continuouslytransmits a radio signal containing the current time and data about itsposition. The time delay between when the satellite transmits a signaland the receiver receives it is proportional to the distance from thesatellite to the receiver. A GPS receiver monitors multiple satellitesand solves equations to determine the precise position of the receiverand its deviation from true time. To get accurate location information,four satellites must be in view of the GPS receiver for it to computefour unknown quantities (three position coordinates and clock deviationfrom satellite time).

GPS based location estimation requires an unobstructed line of sight toat least four satellites to accurately locate the position of the GPSreceiver. Poor connectivity to GPS satellites or connectivity to lessthan 4 GPS satellites leads to inaccuracies in determination of locationbased on GPS. Connectivity to GPS satellites may also be affected byextreme atmospheric conditions, such as geomagnetic storms. Obstaclessuch as walls, buildings, skyscrapers and trees may obstruct the line ofsight of a GPS receiver resulting in inaccurate location estimation. Inareas where a number of obstructions are present, such as in parts ofcities with several obstructions such as skyscrapers or trees, GPSlocation information may be unreliable or inaccurate.

Any discussion of documents, acts, materials, devices, articles or thelike which has been included in the present specification is not to betaken as an admission that any or all of these matters form part of theprior art base or were common general knowledge in the field relevant tothe present disclosure as it existed before the priority date of eachclaim of this application.

Throughout this specification the word “comprise”, or variations such as“comprises” or “comprising”, will be understood to imply the inclusionof a stated element, integer or step, or group of elements, integers orsteps, but not the exclusion of any other element, integer or step, orgroup of elements, integers or steps.

SUMMARY

Some embodiments relate to a system for parking monitoring in an urbanarea, the system comprising:

at least one camera, wherein the at least one camera is positioned tocapture images of the urban area;

a computing device in communication with the at least one camera toreceive the captured images;

the computing device comprising at least one processor and a memoryaccessible to the at least one processor;

wherein the memory comprises a library of reference background imagesand metadata for each reference background image, wherein the metadatacomprises parking location information and parking conditioninformation;

wherein the memory stores program code executable by the at least oneprocessor to configure the at least one processor to:

process a first captured image to determine a licence plate numbercorresponding to a target vehicle in the first captured image;

process a second captured image using a background matching module toidentify a matching reference background image;

determine an identified parking location of the target vehicle and atleast one parking condition based on the metadata of the matchingreference background image;

determine compliance of the target vehicle with the determined at leastone parking condition.

In some embodiments, the first captured image is the same captured imageas the second captured image.

Some embodiments relate to a system for parking monitoring in an urbanarea, the system comprising:

at least one camera, wherein the at least one camera is positioned tocapture images of the urban area;

a computing device in communication with the at least one camera toreceive the captured images;

the computing device comprising at least one processor and a memoryaccessible to the at least one processor;

wherein the memory comprises a library of reference background imagesand metadata for each reference background image, wherein the metadatacomprises parking location information;

the memory also comprises program code executable by the at least oneprocessor to configure the at least one processor to:

process the captured images using a parking indicator detection machinelearning model to identify a parking indicator in at least one of thecaptured images;

on identifying the parking indicator, process the captured images usinga background matching module to identify a matching reference backgroundimage that matches one of the captured images;

determine a parking location based on the metadata associated with thematching reference background image;

determine parking conditions based on the identified parking indicator;

process the captured images to determine a licence plate numbercorresponding to a target vehicle; and

determine compliance of the target vehicle to the determined parkingconditions.

In some embodiments, the licence plate number corresponding to thetarget vehicle is determined using a licence plate detection machinelearning model.

In some embodiments, the parking indicators comprise parking signs orlicence plates and the parking indicator detection machine learningmodel detects parking signs or licence plates in the captured images.

In some embodiments, the memory further comprises parking perimetermetadata associated with each reference background image, and the atleast one processor is further configured to:

process the at least one captured image to identify an image portioncorresponding to the target vehicle in one of the captured images;

determine compliance of the target vehicle to the determined parkingconditions based on the parking perimeter metadata associated with thematching reference background image and the image portion correspondingto the target vehicle.

In some embodiments, the image portion corresponding to the targetvehicle is identified using a vehicle detection machine learning model.

In some embodiments, the background matching module comprises: abackground feature extractor neural network, and the at least oneprocessor is further configured to identify the matching referencebackground image by:

extracting background descriptors from the at least one captured imageusing the background feature extractor neural network;

selecting one or more candidate matching images from the library ofbackground images based on the extracted background descriptors;

performing geometric matching between the at least one captured imageand the candidate matching images to select the matching referencebackground image.

In some embodiments, the geometric matching is performed using a randomsample consensus process.

The parking location information may comprise a street name and a streetnumber; or the parking location information may comprise a street name,a street number and a parking bay identifier; or the parking locationinformation may comprise a longitude coordinate and a latitudecoordinate associated with the parking location.

In some embodiments, the one or more cameras are mounted on asurveillance vehicle,

the computing device is carried by the surveillance vehicle, and

the system further comprises a communication module to enable wirelesscommunication between the computing device and a remote computer system.

Some embodiments are configured to perform parking monitoring in realtime as the surveillance vehicle moves in the urban area.

In some embodiments, the system comprises at least two cameras, with atleast one camera positioned on each side of the surveillance vehicle toperform parking monitoring on both sides of the surveillance vehicle.

In some embodiments, the system comprises at least two cameras, bothcameras are positioned to capture images on one side of the surveillancevehicle; and the background matching module is configured to performbackground matching using captured images from each of the at least twocameras to identify a matching reference background image.

Some embodiments are further configures to track the target vehicleacross the captured images as the surveillance vehicle moves in theurban area.

Some embodiments are further configured to transmit to the remotecomputer system via the communication module one or more of: thedetermined compliance of the target vehicle with the determined parkingconditions; the determined licence plate number corresponding to thetarget vehicle; the determined parking location; or captured images ofthe target vehicle.

In some embodiments, the licence plate detection machine learning modelis configured to identify a portion of the captured image correspondingto a license plate of the target vehicle, and the licence plate numberis determined based on processing the portion of the captured imagecorresponding to the license plate by a character recognition module.

In some embodiments, the parking conditions are determined based oncharacters recognised by processing a portion of the at least onecaptured image corresponding to the identified parking signs using acharacter recognition module.

In some embodiments, at least one reference background image relates toa parking zone start location and another at least one referencebackground image relates to a parking zone end location, anddetermination of compliance of the target vehicle to the identified atleast one parking condition is based on: distance between the identifiedparking location and the parking zone start location; or distancebetween the identified parking location and the parking zone endlocation.

Some embodiments relate to a computer implemented method for parkingmonitoring performed by a computing device comprising at least oneprocessor in communication with a memory, the memory comprising alibrary of reference background images, the method comprising:

receiving images of an urban area captured by a camera in communicationwith the computing device;

processing the captured images using a parking indicator detectionmachine learning model to identify one or more parking indicators in atleast one captured image;

on identifying at least one parking indicator in the at least onecaptured image, process the at least one captured image using abackground matching module to identify a matching reference backgroundimage;

determining a parking location based on the matching referencebackground image;

determining parking conditions based on the determined parking locationor the identified one or more parking indicators;

processing the at least one captured image to determine a licence platenumber corresponding to a target vehicle in the at least one capturedimage; and

determining compliance of the target vehicle to the determined parkingconditions based on the determined licence plate number and thedetermined parking conditions.

Some embodiments relate to a system for location determination, thesystem comprising:

a computing device comprising at least one processor and a memoryaccessible to the at least one processor;

wherein the memory comprises a library of reference background imagesand metadata for each reference background image, wherein the metadatacomprises location information; and wherein the memory stores programcode executable by the at least one processor to configure the at leastone processor to:

receive an input image data from a remote computing device, wherein theinput image data includes image data of at least one image captured bythe remote computing device at a location to be determined;

process the received input image data using a background matching moduleto identify matching reference background image;

determine location information corresponding to the input image databased on the metadata of the matching reference background image in thelibrary; and

transmit the determined location information to the remote computingdevice.

In some embodiments, the background matching module comprises: abackground feature extractor neural network, and the at least oneprocessor is further configured to identify the matching referencebackground image by:

extracting background descriptors from the at least one captured imageusing the background feature extractor neural network;

selecting one or more candidate matching images from the library ofbackground images based on the extracted background descriptors;

performing geometric matching between the at least one captured imageand the candidate matching images to select the matching referencebackground image.

In some embodiments, the geometric matching comprises identifying commonvisual features in the at least one captured image and each of thecandidate matching images.

In some embodiments, the geometric matching is performed using a randomsample consensus process.

In some embodiments, the background feature extractor neural network istrained to extract background descriptors corresponding to one or morestationary features in the at least one captured image.

In some embodiments, the memory stores program code executable by the atleast one processor to further configure the at least one processor to:

receive GPS data corresponding to the input image from the remotecomputing device, wherein the GPS data comprises a low data qualityindicator;

generate a GPS correction signal based on the determined locationinformation;

transmit the GPS correction signal to the remote computing device.

Some embodiments relate to a vehicle mounted system for locationdetermination in an urban area, the system comprising:

at least one camera, wherein the at least one camera is positioned tocapture images of the urban area;

a computing device in communication with the at least one camera toreceive the captured images, the computing device comprising at leastone processor and a memory accessible to the at least one processor, thememory comprising a library of reference background images;

wherein the memory comprises program code executable by the at least oneprocessor to configure the at least one processor to:

extract background descriptors from at least one captured image;

select one or more candidate matching reference images from the libraryof background images based on the extracted background descriptors;

perform geometric matching between the at least one captured image andthe one or more candidate matching reference images to select a singlematching reference background image; and determine a location of thevehicle based on the single matching reference background image.

In some embodiments, the memory further comprises location metadatacorresponding to each reference background image; and the location ofthe vehicle is determined based on the location metadata correspondingto corresponding to the single matching reference background image.

In some embodiments, the background descriptors are extracted from atleast one captured image using a background feature extractor neuralnetwork.

In some embodiments, determining the location is performed in real time.

Some embodiments relate to a computer implemented method for determininga location of a vehicle, the method performed by a vehicle mountedcomputing device comprising at least one processor in communication witha memory, the memory comprising a library of reference backgroundimages, the method comprising:

receiving images of an urban area captured by a camera in communicationwith the computing device;

extracting background descriptors from at least one image captured bythe camera;

selecting one or more candidate matching reference images from thelibrary of background images based on the extracted backgrounddescriptors;

perform geometric matching between the at least one captured image andthe one or more candidate matching reference images to select a singlematching reference background image; and

determine a location of the vehicle based on the single matchingreference background image.

Some embodiments relate to a computer implemented method for determininga location of a vehicle in an urban area, the method performed by avehicle mounted computing device comprising at least one processor incommunication with a memory and at least one camera, the memorycomprising a library of reference background images, the methodcomprising:

capturing an image of the urban area while the at least one camera ismoving in the urban area;

processing the captured image using a background matching module toidentify a matching reference background image;

determining a location of the vehicle based on a metadata of thematching reference background image.

In some embodiments, the at least one camera is mounted on the vehicle.

In some embodiments, the determination of the location of the vehicle isperformed in real time by the vehicle mounted computing device.

Some embodiments relate to a system for location determination in anurban area, the system comprising:

at least one camera, wherein the at least one camera is positioned tocapture images of the urban area while the at least one camera is movingin the urban area;

a computing device moving with the at least one camera and incommunication with the at least one camera to receive the capturedimages;

the computing device comprising at least one processor and a memoryaccessible to the at least one processor;

wherein the memory comprises a library of reference background imagesand metadata for each reference background image, wherein the metadatacomprises location information;

wherein the memory stores program code executable by the at least oneprocessor to configure the at least one processor to:

process a captured image using a background matching module to identifya matching reference background image;

determine a location of the at least one camera and the computing devicebased on the metadata of the matching reference background image.

In some embodiments, processing the captured image using a backgroundmatching module comprises: extracting background descriptors from thecaptured image; selecting one or more candidate matching images from thelibrary of reference background images based on the extracted backgrounddescriptors; performing geometric matching between the captured imageand the candidate matching images to select the matching referencebackground image.

In some embodiments, the background matching module comprises abackground feature extractor neural network configured to extractbackground descriptors corresponding to one or more stationary featuresin the at least one captured image.

In some embodiments, the geometric matching is performed using a randomsample consensus process; and wherein the geometric matching comprisesidentifying common visual features in the at least one captured imageand each of the candidate matching images.

In some embodiments, the computing device is configured to determine thelocation in real-time.

In some embodiments, the at least one camera is mounted on the vehicleto capture images of a vicinity of the vehicle.

In some embodiments, the vehicle is an autonomous driving vehicle.

In some embodiments, the vehicle comprises an on-board GPS receiver andthe vehicle is configured to trigger location determination using thesystem for location determination in response to an image based locationdetermination trigger event.

In some embodiments, the image based location determination triggerevent may comprise at least one of:

or low precision GPS data being generated by the on-board GPS receiver;

crossing of a predefined geo-fence by the vehicle.

Some embodiments relate to a computer implemented method for locationdetermination, the method performed by a computing device comprising atleast one processor in communication with a memory, the methodcomprising:

receiving an input image by the computing device from a remote computingdevice, wherein the input image corresponds to a location to bedetermined;

processing the received input image using a background matching moduleprovided in the memory of the computing device to identify a matchingreference background image from among a library of reference backgroundimages stored in the memory;

determining location information corresponding to the input image basedon the metadata of the matching reference background image; and

transmitting the determined location information to the remote computingdevice.

In some embodiments, the background matching module comprises: abackground feature extractor neural network, and the method furthercomprises identifying the matching reference background image by:

extracting background descriptors from the at least one captured imageusing the background feature extractor neural network;

selecting one or more candidate matching images from the library ofbackground images based on the extracted background descriptors;

performing geometric matching between the at least one captured imageand the candidate matching images to select the matching referencebackground image.

In some embodiments, the geometric matching comprises identifying commonvisual features in the at least one captured image and each of thecandidate matching images.

In some embodiments, the geometric matching is performed using a randomsample consensus process.

In some embodiments, the background feature extractor neural network istrained to extract background descriptors corresponding to one or morepermanent stationary features in the at least one captured image.

In some embodiments, the method further comprises:

receiving GPS data corresponding to the input image from the remotecomputing device, wherein the GPS data comprises a low data qualityindicator;

generating a GPS correction signal based on the determined locationinformation;

transmitting the GPS correction signal to the remote computing device;

wherein the GPS correction signal comprises information accessible bythe remote computing device to determine a more accurate GPS locationdata.

Some embodiments relate to a computer-readable storage medium storinginstructions that when executed by a computer cause the computer toperform the method of parking monitoring according to any one of theembodiments.

Some embodiments relate to a system for parking monitoring in an urbanarea, the system comprising:

at least one camera, wherein the at least one camera is positioned tocapture images of the urban area;

a computing device in communication with the at least one camera toreceive the captured images;

the computing device comprising at least one processor and a memoryaccessible to the at least one processor;

wherein the memory comprises a library of reference background images;

the memory also comprises program code executable by the at least oneprocessor to configure the at least one processor to:

process the captured images using a parking indicator detection machinelearning model to identify one or more parking indicators in at leastone captured image;

on identifying at least one parking indicator in the at least onecaptured image, process the at least one captured image using abackground matching module to identify a matching reference backgroundimage;

determine a parking location based on the matching reference backgroundimage;

determine parking conditions based on the determined parking location orthe identified one or more parking indicators;

process the at least one captured image to determine a licence platenumber corresponding to a target vehicle in the at least one capturedimage; and

determine compliance of the target vehicle to the determined parkingconditions based on the determined licence plate number and thedetermined parking conditions.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of a parking monitoring system according tosome embodiments;

FIG. 2 is a flowchart of a process for parking monitoring according tosome embodiments;

FIG. 3 is a flowchart of a process for location determination accordingto some embodiments;

FIGS. 4 to 7 are examples of captured images illustrating parkingindicator identification and vehicle identification;

FIG. 8 is an example pair of images illustrating a background matchingprocess;

FIG. 9 is an example of a reference background image;

FIG. 10 is an example of a captured image corresponding to the sameparking location as the reference background image of FIG. 9 ;

FIG. 11 is an image obtained by superimposing the image of FIG. 9 on theimage of FIG. 10 ;

FIG. 12 is an image of a portion of a surveillance vehicle according tosome embodiments;

FIG. 13 is a flowchart of a process for image annotation according tosome embodiments;

FIG. 14 is a screenshot an image annotation user interface according tosome embodiments;

FIG. 15 is an image illustrating an outcome of automatic locationextrapolation according to some embodiments;

FIG. 16 a schematic diagram illustrating some aspects of parkingmonitoring according to some embodiments;

FIG. 17 a schematic diagram illustrating some aspects of parkingmonitoring according to some embodiments;

FIG. 18 is a block diagram of a system for location determinationaccording to some embodiments;

FIG. 19 is a flowchart of a process of location determination performedby the system of FIG. 18 ;

FIG. 20 is a block diagram of part of an autonomous vehicle according tosome embodiments;

FIG. 21 is an example pair of images illustrating a background matchingprocess; and

FIG. 22 illustrates an example computer system according to someembodiments.

DETAILED DESCRIPTION

The described embodiments relate to systems and methods for locationestimation using images. In urban areas with a high density ofbuildings, GPS signal connectivity and accuracy are often poor on thestreet or ground level, leading to inaccurate location estimation usingGPS devices. The described embodiments rely on image processingtechniques to determine location information in real-time or nearreal-time.

Some described embodiments also relate to systems and methods formonitoring parking and compliance with parking conditions in urbanareas. Parking spaces are often a limited resource in urban areas. Urbanareas may also have several parking conditions or restrictions in placeto best allocate the limited parking resources while managing theefficiency of traffic. Parking spaces may also be a source of revenuefor municipal organisations. The effectiveness of parking conditions orrestrictions depends on the degree of continuous enforcement. Withoutappropriate enforcement in place, parking conditions may often beviolated, affecting the fair allocation of the parking resources andundermining the efficiency of traffic.

Some embodiments relate to systems and methods for location estimationsuitable for use by or in an autonomous vehicle.

Described embodiments rely on image processing techniques to determinelocation in an urban area by matching persistent background in images.Some embodiments also rely on image processing techniques to identifyobjects such as parking signs, vehicles and licence plates. Someembodiments rely on character recognition techniques to recognisecharacters in parking signs or vehicle licence plates. Some embodimentsalso track identified objects through several images to de-duplicate anyidentified objects and provide more reliable results. Some embodimentsmay be mounted on a surveillance vehicle that goes around an urban areaassessing compliance with parking conditions in real time. Thesurveillance vehicle may include a car, van or other moving vehiclecapable of traversing the streets or roads of the urban area, forexample.

FIG. 1 is a block diagram of a system 100 according to some embodiments.System 100 comprises a surveillance vehicle 110 in wirelesscommunication with a remote computer system 180 over the network 170.The surveillance vehicle comprises at least one camera 120. In someembodiments, the surveillance vehicle may comprise multiple cameras 120.In some embodiments, the camera 120 may be mounted on a roof of thesurveillance vehicle 110. The camera 120 may be at an appropriate angleto the direction of motion of the surveillance vehicle 110 to getsuitable imagery to perform the parking monitoring. In some embodiments,the surveillance vehicle may comprise 6 cameras, 3 cameras facing eachlateral (left and right) side of the surveillance vehicle. The 3 camerason each side may be positioned at an angle of 45-degree, 90-degree, and135-degree respectively to the direction of motion of the surveillancevehicle, for example. The different angles may enable capturing imagesof the urban area from different perspectives. Images from differentperspectives may be used to improve accuracy of the system byredundantly performing the various operations using images fromdifferent angles. Further, some street signage may be parallel to thestreet, other street signage may be perpendicular or at an angle to thestreet. Cameras 120 at different angles may enable capturing streetsignage at various angles. Each camera 120 may capture images at aresolution of 3072×1728 pixels, for example.

System 100 also comprises an image processing computing device 130. Theimage processing computing device 130 is in communication with thecamera 120. Computing device 130 may be positioned inside the vehicle110 or on the vehicle 110 in a secure housing. The computing device 130comprises at least one processor 132, a communication module 134, andmemory 136. Memory 136 may include both volatile and non-volatilememory. In some embodiments, the processor 132 may be specificallydesigned for accelerating the operation of machine learning programs orprocesses. In particular, the at least one processor 132 may comprise agraphics processing unit (GPU) to accelerate execution of machinelearning processes or programs. GPUs enable highly parallel computingoperations and are therefore more suitable for execution of machinelearning processes or programs to obtain results in real-time or nearreal-time. For example, some embodiments may incorporate an NVIDIA™Jetson embedded computing board to provide the processor 132.

Memory 136 comprises a parking indicator detection machine learning (ML)model 142, a reference background image library 144, a vehicle detectionML model 158, a license plate detection ML model 159, a backgroundmatching module 152, a parking condition and compliance determinationmodule 162, and character recognition module 164. The referencebackground image library 144 comprises reference background images 146and metadata associated with the reference background images 148. Thebackground matching module 152 comprises a background feature extractormodule 154 and a geometric match validation module 156. Thecommunication module 134 comprises hardware and software necessary tofacilitate wireless communication between the computing device 130 andnetwork 170. The wireless communication may be achieved through awireless telecommunication network such as a 3G, 4G or 5G network, forexample.

In some embodiments, the vehicle 110 may optionally comprise a GPSreceiver 126 in communication with the image processing computing device130. The GPS receiver 126 may transmit to the computing device 130 GPSData corresponding to the vehicle 110. GPS data generated by GPSreceivers comprises a quality indication of the GPS signal. For exampleGPS data presented in a ‘$GPGGA’(Global Positioning System Fix Data)format stipulated by the National Marine Electronics Association (NMEA)0183 standard comprises a GPS fix quality indicator, a horizontaldilution of precision indicator (HDOP), for example. The imageprocessing computing device 130 may be configured to process the GPSdata generated by the GPS receiver 126 and determine if the one or moreGPS data quality indicators as received point to poor quality GPS signaldata or GPS signal data of less than desired precision. In someembodiments, the GPS signal data with less than desired precision may beused by the background matching module 152 to narrow down the backgroundimages 146 for comparison providing a computational advantage byrelatively narrowing down the number of images in the backgroundmatching module 152 for consideration.

In some embodiments, image processing computing device 130 may not beinside the vehicle 110 or on the vehicle 110 and may be located remotefrom the urban area where location determination is to be performed. Forexample, the image processing computing device 130 may be or form partof a remote server system, such as remote computer system 180. In suchembodiments, the image processing computing device 130 may receiveimages captured by cameras 120 a, 120 b to perform the locationdetermination remotely.

In some embodiments, the image processing computing device 130 maycomprise a GPS correction signal generation module 167 provided in itsmemory 136. Signals generated by conventional GPS receivers are subjectto various forms of errors. The errors may include errors due toobstruction of GPS signals by permanent structures such as buildings ortrees, reflection of GPS signals by buildings, radio interference, solarstorms, for example. One alternative for correction of GPS signal errorsis to use a Differential Global Positioning System (DGPS). A DGPS uses anetwork of fixed ground-based reference stations to broadcast thedifference between positions indicated by the GPS satellite system andknown fixed positions of the ground-based reference stations. A GPSreceiver configured to receive signals from a DGPS system uses thesignals transmitted by the DGPS system to correct its calculatedlocation.

In some embodiments, the GPS correction signal generation module 167generates a GPS correction signal based on a location informationdetermined based on one or more received images. The generated GPScorrection signal may be transmitted by the image processing computingdevice 130 to other computing devices to correct their GPS locations inreal time or near real time. Such other computing devices may includesmartphones or autonomous vehicles or other computing devices that mayuse location data and can benefit from improved location accuracy.

The network 170 may include, for example, at least a portion of one ormore networks having one or more nodes that transmit, receive, forward,generate, buffer, store, route, switch, process, or a combinationthereof, etc. one or more messages, packets, signals, some combinationthereof, or so forth. The network 170 may include, for example, one ormore of: a wireless network, a wired network, an internet, an intranet,a public network, a packet-switched network, a circuit-switched network,an ad hoc network, an infrastructure network, a public-switchedtelephone network (PSTN), a cable network, a cellular network, a fiberoptic network, some combination thereof, or so forth.

Vehicle 110 also comprises a power source 125 for supplying power to thecamera 120 and the image processing computing device 130. The powersource 125 may supply power generated by the vehicle 110's power supplysystems or a battery or a combination of the two.

The parking indicator detection ML model 142, license plate detection MLmodel 159 and the vehicle detection ML model 158 perform objectdetection based on image processing techniques. Various machine learningtechniques may be employed by the embodiments to perform objectdetection. In some embodiments, the parking indicator detection ML model142, license plate detection ML model 159 and the vehicle detection MLmodel 158 may perform object detection using deep learning based neuralnetworks. In some embodiments, object detection may be performed usingthe “you look only once” (YOLO) framework or a “single shot multiboxdetector” (SSD) based model for object detection. In some embodiments,the deep learning based frameworks for object detection may include:regions with convolutional neural networks (R-CNN), or fast region-basedconvolutional network method (Fast R-CNN), or a faster region-basedconvolutional network method (Faster R-CNN).

A CNN as implemented by some embodiments may comprise multiple layers ofneurons that may differ from each other in structure and operation. Afirst layer of a CNN may be a convolution layer of neurons. Theconvolution layer of neurons performs the function of extractingfeatures from an input image while preserving the spatial relationshipbetween the pixels of the input image. The output of a convolutionoperation may include a feature map of the input image. The operation ofconvolution is performed using a filter or kernel matrix and thespecific weights in the filter or kernel matrix are obtained orcalibrated by training the CNN by the processes described subsequently.

After a convolution layer, the CNN in some embodiments implements apooling layer or a rectified linear units (ReLU) layer or both. Thepooling layer reduces the dimensionality of each feature map whileretaining the most important feature information. The ReLU operationintroduces non-linearity in the CNN, since most of the real-world datato be learned from the input images would be non-linear. A CNN maycomprise multiple convolutional, ReLU and pooling layers wherein theoutput of an antecedent pooling layer may be fed as an input to asubsequent convolutional layer. This multitude of layers of neurons is areason why CNNs are described as a deep learning algorithm or technique.The final one or more layers of a CNN may be a traditional multi-layerperceptron neural network that uses the high-level features extracted bythe convolutional and pooling layers to produce outputs. The design of aCNN is inspired by the patterns and connectivity of neurons in thevisual cortex of animals. This basis for design of CNN is one reason whya CNN may be chosen for performing the function of object detection inimages.

The parking indicator detection ML model 142 is configured to detectparking indicators in images captured by camera 120. Parking indicatorsinclude parking signs, or licence plates on vehicles. Parking signsinclude any signage near parking areas that indicate any restrictions onconditions associated with the parking area. The conditions may includea maximum duration or parking permissible for a vehicle during aparticular time period, applicability of any parking fees, any parkingpermit requirements in the parking area, or any other relevant parkingconditions that vehicles in the parking area need to adhere to. Aparking area may be an area adjoining a street designated for parkingvehicles. A parking area may comprise one or more parking bays or slots,with one bay or slot intended for a single vehicle. Parking signs maydefine parking conditions for all bays in a parking area or a subset ofbays in the parking area. Parking signs may also indicate the extent orboundaries of a parking area.

The parking indicator detection ML model 142 is trained to processimages captured by the camera 120 to detect parking indicators. Thetraining of the parking indicator detection ML model 142 may be based ona training dataset of images with parking indicators (ground truthobjects) identified in each image in the training dataset. Using thetraining dataset, the parking indicator detection ML model 142 istrained by supervised learning methodologies such as backpropagation.The trained parking indicator detection ML model 142 identifies parkingindicators in an input image and is also capable of isolating parts ofan input image corresponding to a parking indicator. The trained parkingindicator detection ML model 142 may also be capable of classifyingvarious parking indicators into distinct categories such as a parkingsign or a licence plate. Different cities may have different parkingsigns, and accordingly, the parking indicator detection ML model 142 maybe trained to recognise the parking signs of a particular city, forexample.

The vehicle detection ML model 158 uses similar object detectiontechniques as the parking indicator detection ML model 142 to identifytarget vehicles in images captured by the camera 120. Target vehiclesmay comprise any 2 wheeled vehicles or 4 wheeled vehicles or largervehicles, such as trucks, to which parking conditions may apply. Thevehicle detection ML model 158 is trained to detect one or more targetvehicles in a captured image and identify a portion of the imagecorresponding to the identified vehicle.

The reference background image library 144 comprises referencebackground images 146 and background image metadata 148. Referencebackground images 146 include images that serve as references for thebackground matching module 152 during a process of matching imagescaptured by camera 120 to determine location. Reference background imagemetadata 148 includes metadata regarding each reference backgroundimage. The metadata 148 may include location information associated withthe reference background image. In some embodiments, the locationinformation may comprise longitude coordinate information and latitudecoordinate information. The metadata 148 may include parking conditionsassociated with the parking locations or areas in the vicinity of thelocation associated with each reference background image. The parkinglocation may include a street name, a street number, a parking baynumber, and a city name, for example.

In some embodiments, the background image metadata 148 may compriseinformation defining parking zones. A parking zone may be a continuousarea adjacent to a street designated for parking with a common set ofparking conditions. Background images 146 may comprise multiple imagesof a parking zone. Each parking zone may be defined by a parking zonestart indicator, such as a sign or a post indicating the start of aparking zone. Similarly, the end of each parking zone may be designatedby a sign or a post indicating the end of a parking zone. In someembodiments, the metadata 148 may comprise an indication that a specificbackground image 146 relates to the start of a parking zone or an end ofa parking zone. In some embodiments, the background images 146 may alsoinclude background images beyond the defined parking zone and adjacentto the start and end of the parking zone. Inclusion of background imagesbeyond the defined parking zone allows the estimation of location duringparking monitoring beyond the defined parking zones.

The character recognition module 164 comprises program code thatprocesses images or a portion of an image comprising characters torecognise the characters present in the image. Character recognitioninvolves: pre-processing images to prepare them for recognition, andtext recognition that may be performed by pattern matching oralternatively feature extraction followed by classification of extractedfeatures to recognise text or characters.

The parking condition and compliance determination module 162 performsthe function of determining parking conditions assessing compliance ofvehicles to the determined parking conditions. In some embodiments, theparking conditions may be determined by processing portions of thecaptured images that correspond to parking signs detected by the parkingindictor detection ML model 142 using the character recognition module164. The character recognition module 164 may determine the characterspresent in a parking sign and the recognised characters may be processedby processor 132 executing the program code of the parking conditionsdetermination module 162 to determine the parking conditions indicatedby the parking sign. The determined parking conditions may be, forexample: a limited maximum duration of parking for a particular vehicle,or no parking allowed during certain periods. In some embodiments, theparking conditions determination module 162 may determine parkingconditions based on the location of a parking area or a parking bay in aparking area. The parking conditions may be stored as part of backgroundimage metadata 148 in the reference background image library 144. When acaptured image of a parking location is matched with an image in thereference background image library 144, metadata associated with thematched reference image may be queried to obtain the parking conditionsassociated with the parking area in the captured image. The parkingconditions and compliance determination module 162 may also determinecompliance of a target vehicle to parking conditions based oninformation regarding the target vehicle. The information may include anoutline of the target vehicle. The information may also include atimestamp at which the licence place corresponding to the target vehiclewas first identified in the specific parking area.

The background matching module 152 performs the function of matching animage captured by the camera 120 with an image in the referencebackground image library 144. To efficiently perform backgroundcomparison, in some embodiments the background matching module 152 mayincorporate an image retrieval and comparison process known as DELF(Deep Local Feature) that is based on attentive local featuredescriptors suitable for large-scale image retrieval and comparison.

Before implementing system 100 for parking monitoring, a survey may beconducted of the various parking spaces in an urban area. The survey mayinclude photographing parking areas, and mapping the photographs withthe location they correspond to. The survey essentially populates thecontents of the reference background image library 144. Photographs maybe taken on each parking bay, or parts of a parking area. The need forspecificity of the determined parking location during parking monitoringmay govern the number of photographs required to cover a continuousparking area. For example, if the determined location needs to mapped toa specific parking bay, then a photograph of each parking bay may betaken to populate the reference background image library 144. If thedetermined location could be acceptably within a range of 10 meters,then a photograph may be taken at every 10 meters over a continuousparking area to populate the reference background image library 144. Insome embodiments, the survey to gather reference background images maybe conducted during various lighting conditions or weather conditions toenable determination of location despite changes in lighting or weatherconditions.

After completion of the survey of the parking area, the backgroundfeature extractor module 154 may be trained to process captured imagesto extract background features or background features descriptors in thecaptured images. The background descriptors are an encodedrepresentation of the background of a captured image. The backgrounddescriptors enable comparison of the background of various images usingcomputational optimisation techniques such as a nearest neighboursearch.

The background descriptors may correspond to regions or portions of animage that are permanent or stationary. The determined backgrounddescriptors may accordingly be used to distinguish between stationaryand dynamic portions of an image corresponding to a location. Forexample, in an image of a bus stop with a billboard, the image regionsof the billboard may be considered dynamic and the background featureextractor module 154 may be trained to disregard the portions or regionsof an image that correspond to the billboard. In the same image of thebus stop, portions of the image corresponding to a sign or a permanentstructure such as a post or a shelter can be extracted by the backgroundfeature extractor module 154 as a background descriptor for the image.Accordingly, the background feature extractor module 154 is trained toidentify background descriptors in images that correspond to permanentor stationary structures providing an efficient and reliable basis forcomparison with other images corresponding to the same location.

In some embodiments, the feature extractor module 154 may comprise CNNstrained to extract background feature descriptors from captured images.The CNNs may comprise an additional layer of neurons (attentiondetermination layers) trained to determine attention indicators orweights for features in a captured image. Features corresponding to apersistent background may be given a high attention weight and featurescorresponding to non-persistent or transient background may be given alow attention weight, for example. The attention determination layer inessence helps the background feature extractor module 154 differentiatebetween persistent and transient parts of a captured image and assiststhe efficient comparison of the background of a captured image with thebackground of images in the reference background image library 144.

In some embodiments, the system 100 may perform parking surveillance inreal time or in near real time. Since parking violation may be a verytransient and time specific offence, capturing such violations as theyoccur may be essential for an effective parking condition enforcementstrategy. In an urban area, a very large number of reference backgroundimages may be necessary to canvas the entire parking areas available tovehicles. In addition, the surveillance vehicle 110 may be drivingaround the urban area capturing images. Relatively little compute time(or computing resources) for performing background comparison may beavailable to enable real-time or near real time parking surveillance.The DELF process incorporated by the background matching module 152enables an efficient comparison between a captured image and the imagesin the reference background library 144.

In some embodiments, the background feature extractor module 154 mayidentify more than one matching background image. This may be because ofrelative similarity of backgrounds. In such cases, the geometric matchvalidation module 156 may assist in narrowing down to a single matchedimage. The geometric match validation module 156 may rely on homographytesting processes to narrow down to a single matched referencebackground image. In some embodiments, the geometric match validationmodule 156 may rely on a random sample consensus (RANSAC) process tonarrow down to a single matched reference background image.

The geometric match validation module 156 comprises program code toanalyse two images and determine or identify one or more common visualfeatures in the two images. For example, the geometric match validationmodule 156 may extract distinct visual features or shapes or curves fromtwo images. An overlap between the corresponding distinct visualfeatures or shapes or curves from each of the images may be determinedto assess whether the two images may be considered to be a match andaccordingly may be considered to correspond to a common location.

In some embodiments, the background matching module 152 may rely on ascale-invariant feature transform (SIFT) process to detect and describelocal features in images to perform background matching. In someembodiments, the background matching module 152 may rely on a speeded uprobust features (SURF) process to detect and describe local features inimages to perform background matching.

Remote computer system 180 may be a system for managing actions inresponse to parking violations and managing contents of the referencebackground image library 144, for example deployed in a surveillancevehicle 110 or another computing device. Based on the parking conditionnon-compliance information received from the surveillance vehicle 110,the remote computer system 180 may support or take enforcement actions,such as imposition of fines or alerting authorities regarding theviolation, for example.

The remote computer system 180 comprises at least one processor 174 incommunication with a memory 172. In some embodiment, the memory 172comprises program code to implement a background image annotation tool149.

The background image annotation tool 149 comprises program code toimplement a user interface for annotation of background images 146 andthe management of the background image metadata 148. The backgroundimage annotation tool 149 allows the creation and management of variousversions of the reference background image library 144. A version of thereference background image library 144 created on the remote computersystem 180 may be transmitted to the image processing computing device130 to perform the parking monitoring.

The background image annotation tool 149 comprises an annotation userinterface module 192 that allows a user to review background images 146,annotate the background images 146 and create the background imagemetadata 148 through the annotation process. The background imageannotation tool 149 comprises a location extrapolation module 194 thatallows the extrapolation of locations for background images 146.Annotation and extrapolation using the background image annotation tool149 is described with reference to FIGS. 13, 14 and 15 .

FIG. 2 is a flowchart of a process 200 for parking monitoring accordingto some embodiments. As a precondition to the operation of the parkingmonitoring process of FIG. 2 , a survey of the various parking areas inthe relevant urban area is conducted and the reference background imagelibrary 144 is populated along with the relevant image metadataincluding location data and parking conditions, for example. The parkingindicator detection ML model 142, vehicle detection ML model 158 and thebackground matching module 152 are trained on image data obtained fromthe relevant urban area being monitored.

At 210, camera 120 captures images of an urban area and makes the imageavailable for processing to the processor 132. The images may becaptured as the surveillance vehicle 110 moves through the urban area.Camera 120 may be directed towards the parking areas to get a good viewof parked vehicles and any parking signs. In some embodiments withmultiple cameras 120, images taken at different angles may be madeavailable to the processor 132 for processing. The availability ofimages from different angles enables verification of the various objectdetection and background matching operations performed by the system100.

At 212, the parking indicator detection ML model 142 processes thecaptured image to identify one or more parking indicators in thecaptured image. The parking indicator detection ML model 142 may returnas output coordinates defining a bounding box around each parkingindicator identified in a captured image. The parking indicatordetection ML model 142 may also return as output the nature of theparking indicator, for example whether the parking indicator is aparking sign or a licence plate. In some embodiments, the parkingindicator detection ML model 142 may identify a type of parkingindicator, such as a parking indictor defining a time restriction forparking or a parking indictor signifying a tow away zone. At 214, if atleast one parking indicator is determined in a captured image then theprocess proceeds to 216. If no parking indicators are detected, then theprocess continues at 210, which involves the surveillance vehiclecontinuing to capture images of the urban area as it moves. In someembodiments, steps 212 and 214 may be optional. In embodiments wheresteps 212 and 214 are optional, parking conditions associated with alocation may be determined based on a location estimated by thebackground matching module 152 and by referring to parking conditiondata stored in the background image metadata 148 associated with theestimated location.

At 216, the captured image in which a parking indicator is identified isprocessed by the background matching module 152. The background matchingmodule 152 compares the captured image with the images in the referencebackground image library 146 to identify a single matching referencebackground image. Based on the identified single reference backgroundimage, a parking location is determined by querying the background imagemetadata 148 associated with the single matching reference backgroundimage. The parking location may include a street name, a street number,a parking bay identifier or a parking zone identifier, for example.

In some embodiments, the camera 120 captures a stream of images as itmoves through an urban area. The stream of images may be processed bythe parking indicator ML model 142 to identify parking indicators in theseveral captured images in the stream. The parking indicators detectedin the stream of images may be tracked through the stream of capturedimages using an object tracking image processing technique. Trackingidentified objects in the stream of captured images allows deduplicationof the identified parking indicators. Similarly, the stream of capturedimages may be used for background matching at 216. Using a stream ofimages at 212 and 216 may improve the accuracy of the parking monitoringprocess as potential errors across the stream of capture images may becross checked and verified.

At 218, the parking condition and compliance determination module 162determines the parking conditions associated with the parking locationidentified at 216. In some embodiments, the parking conditions may bedetermined by processing the identified parking indicators includingparking signs using the character recognition module 164 to identify thecharacter or text in the parking signs processing the character todetermine parking conditions. For example, if a parking sign reads “1P730 AM-630 PM” then the parking condition and compliance determinationmodule 162 may determine that a vehicle may be parked in a parking spotfor at most one hour between the time periods 7.30 am to 6.30 μm. Insome embodiments, the character recognition module 164 may also readsymbols such as arrow signs indicating the beginning or end of a parkingarea. The parking condition and compliance determination module 162 mayinterpret such symbols to determine an extent of the area to which theparking conditions apply.

In some embodiments, the parking conditions may be stored as backgroundimage metadata 148 mapped to a particular parking location. The parkingconditions may include a parking perimeter associated with a parkinglocation or a parking bay in a parking location. Some parking conditionsmay be verifiable instantaneously. For example, a vehicle parked in a noparking zone may be instantaneously deemed as being in violation ofparking conditions. Other parking conditions, such as parking for aspecific maximum duration may require more than one observation by thesurveillance vehicle 110 of the same parking area over a period of time.

At 222, the captured image is processed by the license plate detectionML model 159 to detect the licence plate of a target vehicle. Thelicence plate detection ML model may identify a portion of the capturedimage corresponding to a license plate and it may invoke the characterrecognition module 164 to read the characters in a license plate. At222, the process 200 may also record a time stamp associated with thedetection of the license plate of the target vehicle. The time stamp mayinclude a date, and time in a 12 hour or 24 hour time format and a timezone. The recorded timestamp may provide a reference to check compliancewith time based parking conditions.

Steps 224, 226 and 227 are optional in some embodiments wherein thesystem 100 is configured to determine if a target vehicle is present ina parking perimeter defined with respect to a parking location. Steps224, 226 and 227 are performed by the parking condition and compliancedetermination module. At 224, the vehicle detection ML model 158 detectsa target vehicle in the captured image. Detection of the target vehiclemay comprise detection of a portion of the captured image correspondingto the target vehicle, for example. Detection of the target vehicle maycomprise detection of an outline corresponding to the target vehicle,for example. At 226, the parking conditions and compliance determinationmodule 162 may retrieve a parking perimeter associated with theidentified parking location and determine if the identified targetvehicle is positioned within the parking perimeter. If the identifiedtarget vehicle is at least partially not positioned within the parkingperimeter, then the parking conditions and compliance determinationmodule 162 may identify a noncompliance to the determined parkingconditions. The image processing technique of determining if theidentified target vehicle is positioned within the parking perimeter maybe referred to as polygon warping. Polygon warping may comprisecalculation of a transformation matrix to superimpose a parkingperimeter on a capture image. The transformation matrix maps a pixel inthe captured image with a pixel in the matched reference backgroundimage. The matrix transformation may include procrustes analysis, orperspective transformation, for example.

At 227, the parking conditions and compliance determination module 162may query previous observations of the target vehicle at the sameparking location. If time restricted parking conditions apply to theparking location, then the parking conditions and compliancedetermination module 162 based on previous observations may estimate theduration since the first observation of the target vehicle at theparking location. Based on the calculated duration since the firstobservation, the parking conditions and compliance determination module162 may determine if the target vehicle is in compliance with any timebased parking conditions.

At 228, the parking conditions and compliance determination module 162determines the compliance of the target vehicle to the parkingconditions determined at 218. Determination of compliance at 228 may bebased on the license plate detected at 222, and any previousobservations of the same licence plate at the same location.Determination of compliance at 228 may also be based on the output ofany one or more of steps 224, 226 and 227. Determination of complianceat 228 may result in a non-compliance determination, in which case theparking condition and compliance determination module 162 generates anon-compliance outcome, for example in the form of a flag, object orpacket for remote or local transmission.

At 229, the computing device 130 transmits the outcome of the compliancedetermination at 228 to the remote computer system 180. If anon-compliance determination is made at 228, the outcome transmissionmay include one or more captured images showing con-compliance, thelicense plate number of the target vehicles, the time stamp(s) at whichthe target vehicle was detected and the parking condition(s) that thetarget vehicle was in violation of. The non-compliance outcometransmission may include sufficient evidentiary details to allowsanctions for noncompliance with parking conditions to be imposed by arelevant authority, for example. The non-compliance outcome transmissionmay occur immediately after a non-compliance is determined to enforcethe parking conditions in real-time or in near real-time. For example,the non-compliance outcome transmission may be routed to a local clientdevice carried by a nearby compliance officer so that the officer cantake an appropriate enforcement action in relation to the vehicle.Alternatively or in addition, the non-compliance outcome transmissionmay be routed to a server device, such as remote computer system 180 vianetwork 170, in order for that server device to record thenon-compliance and issue a suitable notification and/or fine to theregistered owner of the non-compliant vehicle.

FIG. 3 is a flowchart of a process for location determination 300according to some embodiments. In some embodiments, the process oflocation determination 300 may be performed by the image processingcomputing device 130 in communication with the camera 120 a mounted onthe surveillance vehicle 110. In some embodiments, the process oflocation determination 300 may be performed by a computing device 130not provided in the surveillance vehicle 110 and configured to receivean input image or have access to an input image for locationdetermination.

At 310, the camera 120 mounted on the vehicle 110 captures images of theurban area and makes the images available to the processor 132. In someembodiments, step 310 may comprise the computing device 130 notpositioned in the surveillance vehicle receiving captured images of theurban area from a remote computing device or a remote camera 120.

At 312, the processor 132 processes a captured image to determinebackground descriptors associated with the image using the backgroundfeature extractor module 154. The background descriptors are in essencethe persistent distinguishing aspects of the captured image thatuniquely identify the background present in the captured image. In someembodiments, the background descriptors may be extracted by applying afully convolutional network (FCN) constructed by using the featureextraction layers of a convolutional neural network (CNN) trained with aclassification loss. The background feature extractor module 154 maycomprise a classifier with attention parameters to explicitly measurerelevance of background descriptors and it may be trained to distinguishbetween persistent background features and transient backgroundfeatures. Persistent background features may include distinctive partsof a building in the background or a permanent light post or bollard,for example. Transient background features may include billboards orhoarding that are subject to change, for example. As an output of step312, a set of background descriptors or features are obtained for thecaptured image. The background descriptors or features may berepresented in the form of a vector or matrix. As a result of step 312,the captured image (that may be a high resolution image) may betransformed into a compressed or succinct vector or matrix of relevantbackground descriptors or features.

As the number of images in the reference background image library 144can be quite large (for example 10,000 to 1 million images), use ofconventional image comparison techniques may not provide a substantiallyaccurate or a substantially computationally efficient approach forcomparing the image captured or received at 310 with each of the imagesin the reference background image library 144. The backgrounddescriptors extracted at step 312 represented as a succinct vector ormatrix of relevant background descriptors or features in the capturedprovide a computationally efficient basis for comparison with the imagesin the reference background image library 144. The backgrounddescriptors extracted at step 312 provide an effective encoding of asubset of features relating to persistent or stationary backgroundfeatures while disregarding or excluding or deprioritising features inthe captured image not relevant to the comparison with images in thereference background image library 144. Features in the captured imagenot relevant to comparison with images in the reference background imagelibrary 144 may include image portions or features relating toindividuals, billboards or dynamic signage, for example.

At 314, a search is performed to select one or more candidate matchingimages from the library of reference background images 144. In someembodiments, the search may be a nearest neighbour search implemented bya combination of KD-tree and product quantisation (PQ). The search isperformed using the pre-determined background descriptors or features ofthe images in the library of reference background images 144. An outputof 314 is a set of candidate reference background images that are apotential match for the captured image. The points of the captured imagethat correspond to points in the candidate matching images may bereferred to as key points. The number of key points identified between acaptured image and an image in the reference background image library144 may be considered as a factor in determining the similarity of thebackground in the two images.

At 316, geometric matching or verification is performed between thecaptured image and the candidate reference background images. Geometricverification includes a more detailed comparison between two images toassess if they share a common background. With the reduced number ofcandidate reference background images obtained at step 314, thegeometric matching step is not as computationally intensive incomparison to performing geometric matching across the entire referencebackground image library 144. In some embodiments, the geometricverification may be performed using a random sample consensus (RANSAC).The outcome of 316 may be the confirmation of a single candidatereference image as a match for the capture image. As part of thegeometric match validation, a matching number of points of interest maybe identified between the captured image and each of the candidatereference background images. Each point of interest in the capturedimage may correspond to a stationary or permanent background in theimage. The number of matching points of interest between two images maybe used as a metric to quantify a degree to which the two images match.For example, images A and B with 30 matching points of interest may be abetter match than images A and C with 10 matching points of interest.The candidate reference background image with the highest number ofmatching points of interest may be considered the closest match to thecaptured image. A point of interest match threshold of the minimumnumber of matching points may be used to establish a minimum degree ofgeometric matching. If no candidate reference background image comprisematching points of interest above the point of interest match threshold,then the captured image may be considered as not matched. In someembodiments, the point of interest match threshold may be 5 points, 10points, 15 points, 20 points, or 25 points, for example. FIGS. 8 and 21illustrate examples of matching points of interest between two images.At 318, the location of the vehicle may be determined by querying thelocation metadata associated with the matched reference backgroundimage.

FIG. 4 is an example of a captured image 400 showing identified parkingindicators after step 212 of flowchart 200. The identified parkingindicators include license plates 440 and 420. Image 400 alsoillustrates identified vehicles 410 and 430. The vehicles and licenseplates are surrounded by bounding boxes indicating the portions of thecaptured image that correspond to the vehicles and license platesrespectively.

FIG. 5 is an example of a captured image 500 showing identified parkingindicators after step 212 of flowchart 200. In image 500, parkingindicators 510 and 520 corresponding to parking signs have beenidentified by the parking indicator detection ML model 142. Similarly,FIG. 6 is an example of a captured image 600 wherein parking indicators630, 620 and vehicle 610 have been identified by the parking indicatordetection ML model 142 and vehicle detection ML model 158 respectively.FIG. 7 is an example of a captured image 700 in which parking indicator710 has been identified by the parking indicator detection ML model 142.

FIG. 8 is an example image 800 illustrating a stage in the process ofbackground matching performed by the background matching module 152.Image 800 comprises two parts, image 810 and image 820. Image 810 is areference background image stored in the reference background imagelibrary 144. Image 820 is an image captured by camera 120 during thecourse of parking surveillance in an urban area. The various parts of apersistent background in image 820 provide a basis for comparison withthe image 810 and enable the determination of a parking location basedon the location metadata associated with reference image 810. Forexample, feature 816 (part of a light post) in image 810 corresponds tothe identical feature 826 in the captured image 820. Feature 812 (partof parking sign post) in image 810 corresponds to the identical feature822 in the captured image 820. Feature 814 (part of building in thebackground) in image 810 corresponds to the identical feature 824 in thecaptured image 820. Feature 818 (part of roadside curb) in image 810corresponds to the identical feature 828 in the captured image 820.

FIG. 9 is an example of a reference background image 900. Metadata ofthe reference background image 900 also comprises a perimeter 910 thatdefines an appropriate area for parking.

FIG. 10 is an example of a captured image 1000 in the locationcorresponding to the reference image 900. FIG. 10 illustrates a targetvehicle 1010.

FIG. 11 is an image 1100 obtained by superimposing the reference image900 on the captured image 1000. As is discernible from image 1100, thesuperimposing allows the verification by the parking condition andcompliance determination module 162 of the compliance of the vehicle1010 with the parking condition of the parking perimeter 910.

FIG. 12 is an image of part of a surveillance vehicle 1210 illustratinga vehicle-mounted system comprising cameras 120 according to someembodiments. The surveillance vehicle 1210 of FIG. 12 has 4 cameras1221, 1222, 1223, and 1224, for example, all of which are provided in asecure external housing 1229. In other embodiments, 6 or 8 or morecameras may be mounted in the external housing 1229. External housing1229 may have a generally weatherised hard shell exterior that definesmultiple imaging windows (or imaging apertures) at spaced locationsaround a lower periphery of the external housing 1229. Each of thecameras 120 (e.g. cameras 1221, 1222, 1223, 1224) is positioned tocapture images of different areas around the vehicle 1210 through arespective imaging window of the external housing 1229. Two cameras facethe front of the vehicle and two cameras face the rear of the vehicle,for example. The cameras are appropriately angled to get the best angleof vision towards parking areas by the side of the street on which thesurveillance vehicle 1210 could drive.

The vehicle 1210 also comprises a computer housing 1227 inside which thecomputing device 130 may be located. The computer housing 1227 isdisposed inside the secure external housing 1229. The computer housing1227 and cameras 120 are protected by the secure external housing 1229,which generally encloses the computer housing 1227 and cameras 120.Secure external housing 1229 and computer housing 1227 may beweatherproof to secure and protect the operation of the cameras 120 andthe computing device 110 from tampering and in unfavourable weatherconditions, such as during rain or snow. An internal cavity of thesecure external housing 1229 may be manually lockable and openable foreasy access to update or replace the various components in the secureexternal housing 1229. An internal cavity of the computer housing 1227may be manually lockable and openable for easy access to update orreplace the various computing components in the computer housing 1227.The secure external housing 1229 may be fixed on the vehicle 1210 usingone or more clamps to a roof rack 1233, for example.

In FIG. 12 , cameras 120 point at an angle of approximately 45 degreesto the direction of travel (front and rear directions) of the vehicle1210. In some embodiments, the camera 120 may point at an angle ofapproximately 60 degree or 90 degree to the direction of travel ofvehicle 1210. Multiple cameras 120 arranged to cover different fields ofview allow the capture of images of a parking location from differentperspectives. Parking locations may be subjected to various kinds ofobfuscation by vehicles and obtaining images from different anglesallows the comparison with the images in the reference background imagelibrary 144. Using multiple cameras 120 to perform background matchingimproves the accuracy and reliability of the background matching module152 by introducing redundancies and enabling verification of matchingusing images captured from different perspectives.

FIG. 13 is a flowchart of a process for image annotation 1300 accordingto some embodiments. The process for image annotation 1300 may beperformed using the background image annotation tool 149. The process ofimage annotation 1300 allows the creation of the reference backgroundimage library 144, which forms the basis of location estimation by theimage processing computing device 130. Annotation information generatedthrough the process 1300 forms part of the background image metadata148.

At 1310, images of the area where the parking monitoring is to beperformed are captured. The images may be captured using a cameramounted on a vehicle driven around the area where the parking monitoringis to be performed. The area may include a large urban area extending toseveral or tens of square kilometres with parking areas located atintervals along, or extending along, lengths of several kilometres, forexample. Step 1310 involves surveying the entire urban area whereparking monitoring is to be performed. Images are captured by thevehicle 110 in a sequence defined by the direction of motion of thevehicle 110 as it captures the images. Sequence information associatedwith the captured images is retained to assist the subsequent steps ofannotation.

Parking zones in urban areas are often defined by signs or posts orindicators located adjacent to the parking zone or parking area on thesidewalk. The signs or posts may indicate a start or an end of a parkingzone, thereby demarking the beginning or end of a zone. Specific parkingconditions may be associated with a parking zone. For the purpose ofparking monitoring, determination of a location as inside or outside aparking zone allows the assessment of compliance of vehicles to theparking conditions.

At 1312, an image corresponding to the start of a parking zone isannotated using the annotation user interface 192. The annotationcomprises assigning a location to the image corresponding to the startof the parking zone and identifying that location as the start of aparking zone. The location may be assigned by indicating longitude andlatitude coordinates of the location corresponding to the start of theparking zone.

Similarly at 1314, an image corresponding to the end of a parking zoneis annotated using the annotation user interface 192. The annotationcomprises assigning a location to the image corresponding to the end ofthe parking zone and identifying that location as the end of a parkingzone. The location may be assigned by indicating longitude and latitudecoordinates of the location corresponding to the end of the parkingzone.

An image (or plurality of images) corresponding to a location of thestart of a parking zone and another image (or plurality of images)corresponding to a location of the end of a parking zone may togetherdefine an extent of a specific parking zone with its specific parkingconditions. In some embodiments, the defined parking zone may correspondto official parking zones defined by municipalities and the annotationprocess may comprise designation of a municipal parking zone code to thedefined parking zone. The designation of a municipal parking zone codemay allow extraction of parking conditions associated with the definedparking zone from a municipal parking conditions database remote to theimage processing computing device 130, for example.

Since the images captured at 1310 comprise sequence information, byidentifying and annotating the start and end of a parking zone, theimages between the images associated with the start and end of a parkingzone may be automatically assigned to the parking zone defined by thestart and end of the parking zone. In some embodiments, the assignmentof the images to a defined parking zone may be assisted by annotation ofone or more images sequenced between the images defining the start andend of a parking zone. Some parking zones may extend through, across oraround street corners, and it may be advantageous to annotate imagescorresponding to street corners to assist the automatic extrapolation orallocation of images in a sequence to a defined parking zone. Atoptional step 1316, images sequenced between the image corresponding tothe start and end of a parking zone may be annotated. The annotation mayinvolve assignment of the image to a defined parking zone and assignmentof a location to the image.

At 1318, the location extrapolation module 194 processes the annotationsat steps 1312, 1314 and 1316 to automatically annotate the sequence ofimages between the images corresponding to the start and end of thedefined parking zone. The automatic annotation may involve automaticallocation of extrapolated locations to the images sequenced between theimages corresponding to the start and end of a parking zone. In someembodiments, the automatic annotation may involve calculation of adistance between the locations corresponding to the start and end of aparking zone, division of the distance by the number of images sequencedbetween the start and end of the parking zone and an incrementalallocation of location coordinates to each image sequenced between thestart and end of a parking zone.

For example, if the distance between the start and end of a parking zoneis 9 m, and there are 8 images between the start and end of the parkingzone, then each image may be assigned a location separated by 1 m on astraight line defined by the locations corresponding to the start andend of a parking zone. In embodiments where a parking zone extendsthrough a street corner, the images annotated at 1316 corresponding to astreet corner may assist the location extrapolation process. In someembodiments, the location extrapolation module 194 may rely on apublicly available street routing Application Programming Interface(API) such as the Google™ Roads API to assist the automaticextrapolation of location information based on the location informationassociated with the start and end of a parking zone.

Through the steps 1312 and 1318, the sequence of images captured at 1310are annotated by assignment of a location and a parking zone. At 1320,the annotated images are included in the reference background imagelibrary 144 to enable location estimation on the basis of the annotationinformation generated through the steps 1312 and 1318. In embodimentswhere process 1300 is performed on a computer system remote to the imageprocessing computing device 130, the annotated images may be transmittedto the image processing computing device 130 along with the annotationinformation for inclusion in the background image metadata 148.

FIG. 14 illustrates a screenshot 1400 of an image annotation userinterface 192 according to some embodiments. A panel 1410 in the userinterface lists a series of files names of a sequence of images forannotation. Clicking on an individual file name in panel 1410 presentsthe image in panel 1420. Panel 1430 illustrates a Google™ street viewinterface. The Google™ street view interface 1430 provides an efficientmechanism for annotation of a location of the mage presented in panel1420. In alternative embodiments, other similar mechanisms for locationannotation may be used. The Google™ street view interface of panel 1420is clickable and navigable to obtain a desirable location to annotatethe image in panel 1420. Clicking on a specific part of panel 1430 dropsa pin on the Google™ street view location and retrieves the coordinatesassociated with the location where a pin is dropped. The retrievedlocation may comprise longitude information and latitude informationwith relatively high accuracy. After clicking on the panel 1430 to dropa pin to select a location, the button 1450 may be clicked to annotate aselected image to a location selected by dropping a pin in panel 1430.Buttons 1450 may by clicked to navigate through the various images andannotate them. As discussed in relation to process 1300 of FIG. 13 , itis not necessary to annotate each image. With annotations indicating thestart and end of a parking zone, it may be possible to extrapolate thelocations of the rest of the images using the sequence information andpublicly available information regarding the layout of roads such as theGoogle™ Roads API.

In some embodiments, location annotation may rely on common stationaryor permanent structures visible in both the Google™ street view panel1430 and the image in panel 1420. As illustrated in screenshot 1400, theimage in panel 1420 and the Google™ street view location in panel 1430have a post 1470. It is advantageous to annotate locations using a baseof a permanent structure such as the post 1470. As illustrated inscreenshot 1400, a marker 1440 for annotation is positioned at the baseor bottom of the post 1470 at sidewalk or footpath level. Positioningmarkers for annotation at the bottom of permanent structures such asposts improves the efficiency of automatic extrapolation of locationbecause conventionally permanent structures such as posts are located ata relatively uniform distance from the street. Since permanentstructures are located at a relatively uniform distance from the street,using the foot or base of such permanent structures for annotationprovides a more systematic and consistent basis for annotation.

FIG. 15 is an image 1500 illustrating an outcome of automatic locationextrapolation according to some embodiments. Image 1500 is a mapcorresponding to an area where image annotation has been performedaccording to the process 1300 of FIG. 13 . Markers 1510 and 1530correspond to locations that have been identified as a start of aparking zone and an end of a parking zone respectively. Markers 1520 and1522 correspond to locations that have been automatically assigned toimages based on an extrapolation of locations 1510 and 1530corresponding to the start and end of a parking zone.

FIG. 16 is a schematic diagram 1600 illustrating some aspects of parkingmonitoring according to some embodiments. Diagram 1600 illustrates atarget vehicle 1680 and the surveillance vehicle 110. The target vehicle1680 is located in a parking zone defined by a parking zone startlocation 1610 and a parking zone end location 1620. The surveillancevehicle 110 has the camera 120 mounted on it which has a line of sightillustrated by 1640. Using the image captured by camera 120, a locationcoordinate corresponding to point 1650 is determined using step 216 offlowchart 200 of FIG. 2 . Using the determined location information ofpoint 1650, the image processing computing device 130 is able toretrieve the coordinates corresponding to the locations 1610 and 1620.Location 1610 corresponds to the start of a parking zone and location1620 corresponds to an end of the parking zone. The coordinatescorresponding to the locations 1610 and 1620 may be stored in thebackground image metadata 148 of the reference background image library144.

Using the coordinates of points 1610 and 1650, a distance Di (distancebetween start of parking zone and the identified location) is determinedby the parking compliance determination module 162. Similarly, adistance De (distance between end of parking zone and the identifiedlocation) may be determined using coordinates of points 1650 and 1620.In the illustrated position, the image processing computing device 130also determines a licence plate associated with the target vehicle 1680using the licence plate detection machine learning model 159. A distanceDo (offset distance) is a predetermined offset distance between alocation corresponding to the line of sight 1640 and a determinedlicence plate of the target vehicle 1680. The Do value may depend on theangle at which the camera 120 or cameras 120 a, 120 b are positioned;and the conditions of the urban environment itself such as the size ofthe parking lots. The offset distance is predetermined based onexperimentation in an urban area. A determined offset distance is fixedfor an urban area or a subset of a region within an urban area withcommon parking conditions.

In some embodiments, camera 120 a may be positioned at an angle of 90degrees to the direction of motion of the surveillance vehicle 110 andmay be pointed towards the adjacent sidewalk; and camera 120 b may bepositioned at an angle of 45 degrees to the direction of motion. In suchembodiments, images from camera 120 a may be used for locationestimation and images from camera 120 b may be used for licence platedetection. In such embodiments, the offset distance (Do) may range from3 m to 8 m, for example.

A distance (Ds) of the target vehicle 1680 from the start of the parkingzone (1610) may be determined by adding Di with Do as illustrated in theFIG. 16 . The total length of the parking zone (Dz) may be determined bydetermining the distance between 1620 and 1610. If Ds<Dz, then thetarget vehicle 1680 may be considered to be parked in the identifiedparking zone. If Ds>Dz, then the target vehicle 1680 may be consideredto be parked outside the parking zone.

FIG. 17 is another schematic diagram 1700 illustrating some aspects ofparking monitoring according to some embodiments. Diagram 1700illustrates the parking zone of FIG. 16 defined by the points 1610 and1620. The reference background image library 144 comprises referenceimages not just from within the parking zone but also reference imagesoutside of the parking zone to allow the surveillance vehicle 110 todetermine its location even when the line of sight 1740 of camera 120 isdirected to an area outside of a parking zone.

In the schematic diagram 1700, the line of sight of camera 120 a isdirected to location 1750. Using the image captured by the camera 120 a,coordinates of the location 1750 may be determined by the backgroundmatching module 152. At the same time, a licence plate corresponding tothe target vehicle 1780 may be detected by the image processingcomputing device 130 using the licence plate detection machine learningmodel 159. Based on the determined location coordinates of location1750, the coordinates of the parking zone start location 1610 and theparking zone end location 1620 may be determined by the image processingcomputing device 130 by referring to the background image metadata 148.With the coordinates of points 1610, 1620 and 1750 determined, thedistance Di between 1610 and 1750 is calculated. As discussed withrespect to FIG. 16 , the offset distance Do is pre-determined. The sumDi+Do is determined to obtain a location 1730 corresponding to thetarget vehicle 1780. Based on the determined location 1730, a distanceDs between the target vehicle 1780 and start of the parking zone 1610 isdetermined. Similar to FIG. 16 , distance Dz between the start and endof the parking zone is determined. If Ds>Dz, then the target vehicle1780 may be considered to be positioned outside the parking zone. IfDs<Dz, then the target vehicle 1780 may be considered to be parkedinside the parking zone. As illustrated in FIG. 17 , Ds>Dz. Accordingly,the target vehicle 1780 is determined to be positioned outside theparking zone.

FIG. 18 illustrates a block diagram of a system 1800 for locationdetermination according to some embodiments. The system 1800 isconfigured to receive an input image and GPS location data from a remotecomputing device 1810. The remote computing device 1810 may be ahandheld computing device, such as a smartphone, a tablet device oranother portable computing device that uses GPS location data. Theremote computing device 1810 may include or form part of an on-boardcomputer system of a vehicle, such as an autonomous passenger or workvehicle, for example. In response to the input image and the GPSlocation data, the image processing computing device 130 is configuredto generate a GPS correction signal and transmit the GPS correctionsignal to the remote computing device 1810. The remote computing device1810 comprises at least one processor 1810 in communication with amemory 1840. The remote computing device 1810 also comprises a camera1820 and a GPS receiver 1830. The memory 1840 comprises computer programcode to implement a location determination module 1850 and in someembodiments a background feature extractor module 1855.

The location determination module 1850 is configured to process the GPScorrection signal received from the image processing computing device130 and determine a corrected location. The remote computing device 1810may be a smartphone or a computing device used for a location survey,for example. In environments with limited or inaccurate access to GPSdata, the remote computing device 1810 captures one or more images ofits location using camera 1820 and transmits the one or more images tothe image processing computing device 130 over the network 170. In someembodiments, the remote computing device 1810 may process the capturedimages using the background feature extractor module 1855 to extract avector or matrix of features corresponding to persistent or stationarybackground objects in each of the captured images. The use of the vectoror matrix of features corresponding to persistent or stationarybackground objects in the captured images is a way of compressing theimage data to a size less than the size of the originally capturedimages. Instead of transmitting the full data of the captured images tothe image processing computing device 130, the remote computing device1810 may transmit the captured image data vector or matrix of featurescorresponding to persistent or stationary background objects in thecaptured images to economise the amount of data transmitted whileretaining sufficiently detailed data regarding persistent or stationarybackground objects in the captured image.

The remote computing device 1810 may also transmit the GPS data obtainedfrom the GPS receiver 1830 to the image processing computing device 130.In response to the transmitted one or more images, the image processingmay return to the remote computing device 1810 location informationcorresponding to the one or more images. In some embodiments, the imageprocessing computing device 130 may also return a GPS correction signalto the remote computing device 1810.

FIG. 19 illustrates a flowchart of a method 1900 of locationdetermination performed by the system 1800 of FIG. 18 . The method 1900includes the steps 312, 314, 316, and 318 of method 300 of FIG. 3 . Atstep 1910, the image processing computing device 130 receives one ormore images (or a vector or matrix of features corresponding topersistent or stationary background objects for each captured image)from the remote computing device 1810. At step 1910, the imageprocessing computing device 130 may also receive GPS data from theremote computing device 1810. The received GPS data may be poor qualityGPS data obtained by the remote computing device 1810 that the computingdevice 1810 is seeking to correct.

At step 312, as described with reference to FIG. 3 , the imageprocessing computing device 130 extracts background descriptors from oneor more images received at step 1910. The extracted backgrounddescriptors may be represented in the form of a vector or a matrix. Atstep 314, one or more candidate matching images are selected from thereference background image library 144 by the background matching module152. At step 316, the geometric match validation module 156 performs ageometric match validation and determines a single matching referencebackground image from the background image library 144. At 318, thelocation metadata of the matching reference background image is checkedto obtain location information corresponding to the one or more imagesreceived at step 1910.

In some embodiments, an optional step 1920 of generation of a GPScorrection signal is performed. The GPS correction signal may begenerated on the basis of the location information identified at step318 and the GPS signal from the remote computing device 1810 received atstep 1910. The GPS correction signal may encode information that may beprocessed by the location determination module 1850 of the remotecomputing device 1810 in order to make an adjustment to the GPS locationdata obtained from the GPS receiver 1830 to obtain accurate locationinformation. At step 1930, the location information determined at step318 and the GPS correction signal determined at 1920 may be transmittedto the remote computing device 1810.

As illustrated by the system 1800 of FIG. 18 and the flowchart 1900 ofFIG. 19 , some embodiments relate to the image processing computingservice receiving images from a remote computing device, determininglocation information based on the images and transmitting locationinformation or a GPS correction signal in response to the receivedimages. Accordingly, some embodiments address inaccuracy or poor qualityof GPS signals in certain environments by providing alternative methodsand systems for location determination based on visual informationcaptured in images.

FIG. 20 is a block diagram of part of an autonomous vehicle 2000provided with the image processing computing device 130 according tosome embodiments. The autonomous vehicle 2000 comprises an autonomousvehicle computing device 2030 to perform the various operations tocontrol the driving operations of the autonomous vehicle 2000. In someembodiments, the autonomous vehicle computing device 2030 is incommunication with the image processing computing device 130, and insome embodiments, the components and functions of the image processingcomputing device 130 are subsumed in and form part of the autonomousvehicle computing device 2030. The autonomous or self-driving vehicle2000 may rely on a combination of the GPS receiver 126 and the imageprocessing computing device 130 for determining its location. Theautonomous vehicle 2000 may switch to location determination based onthe image processing computing device 130 in urban environments and/orareas where the quality or precision of the signals received by the GPSreceiver is less then optimal for autonomous driving. An image basedlocation determination trigger event may initiate an operational mode inwhich the autonomous vehicle 2000 frequently makes determinations of itslocation using the image processing computing device 130. The triggerevent may include poor quality or low precision data received by the GPSreceiver 126 or the vehicle 110 passing through a predefined geo-fencedesignating an area where image based location estimation may be morereliable. In some embodiments, parts of the image processing andlocation estimation computational capabilities may be implemented on theremote server system 2040 and the autonomous vehicle computing device2030 may be configured to wirelessly communicate with the remote serversystem 2040. The wireless communication may be performed through acellular network such as a 3g, 4g or 5g cellular network for example.

FIG. 21 is an example image pair 2100 illustrating a stage in theprocess of background matching performed by the background matchingmodule 152. Image 2100 comprises two parts, a first image 2130 and asecond image 2140. Image 2140 is a reference background image stored inthe reference background image library 144. Image 2130 is an imagecaptured by camera 120 during the course of parking surveillance in anurban area. The various parts of a background in image 2140 provide abasis for comparison with the image 2130 and enable the determination ofa parking location based on the location metadata associated withreference image 2140. It is assumed that each captured image 2130 willhave various static image features that are also present in a referenceimage 2140 taken at substantially the same location, even if variousdynamic image features will vary over time. For example, feature 2112(part of a light post) in image 2140 corresponds to the identicalfeature 2102 in the captured image 2130. Feature 2114 (part of a foot ofa letter box) in image 2140 corresponds to the identical feature 2104 inthe captured image 2130. Feature 2116 (part of a façade of a building)in image 2140 corresponds to the identical façade feature 2106 in thecaptured image 2130. Feature 2118 (a textured grate or part of thefootpath) in image 2140 corresponds to the identical grate or footpathfeature 2108 in the captured image 2130. Feature 2120 (anon-dynamic/static part of a signboard) in image 2140 corresponds to theidentical feature 2110 in the captured image 2130. Unlike the images ofFIG. 8 , the images of FIG. 21 have been captured at an approximately 90degree angle to the direction of travel of the surveillance vehicle 110.

During experimentation, accuracy of the embodiments was evaluated tobenchmark the performance of the embodiments. Accuracy was measuredbased on an accuracy of the location determined at step 216, or step 318using a margin of error of 0.5 m to 2 m, for example. For someembodiments, the accuracy of location determined was in the order of 95%or greater when location estimation was performed over an urban areaextending over a length of 30 km, 40 km, 50 km, 60 km or 70 km, forexample. The accuracy performance of 95% or greater was obtained in bothdaytime and night time conditions and over different seasons of theyear, including winter and summer.

The various models and modules of computing device 130 including theparking indicator machine learning model 142, background matching module152, licence plate detection machine learning model 159, vehicledetection machine learning model 158 may be or comprise program code,libraries, Application Programming Interfaces (APIs), metadata,configuration data, dependencies, frameworks and other necessary codecomponents or hardware components to implement the functionality of thevarious modules or models. The various machine learning models orcomponents may incorporate alternative machine learning methodologiesincluding: supervised, unsupervised, semi-supervised or reinforcementlearning based methodologies, for example. The various machine learningmodels or components may incorporate one or more components from machinelearning frameworks including: OpenCV, TensorFlow, PyTorch, Caffe,EmuCV, VXL, GDAL, MIScnn, Marvin, and Kornia, for example.

FIG. 22 illustrates an example computer system 2200 according to someembodiments. In particular embodiments, one or more computer systems2200 perform one or more steps of one or more methods described orillustrated herein. In particular embodiments, one or more computersystems 2200 provide functionality described or illustrated herein. Inparticular embodiments, software running on one or more computer systems2200 performs one or more steps of one or more methods described orillustrated herein or provides functionality described or illustratedherein. Particular embodiments include one or more portions of one ormore computer systems 2200. Herein, reference to a computer system mayencompass a computing device, and vice versa, where appropriate.Moreover, reference to a computer system may encompass one or morecomputer systems, where appropriate. Computing device 130 is an exampleof computer system 2200. Computing device 1810 is another example ofcomputer system 2200.

This disclosure contemplates any suitable number of computer systems2200. This disclosure contemplates computer system 2200 taking anysuitable physical form. As example and not by way of limitation,computer system 2200 may be an embedded computer system, asystem-on-chip (SOC), a single-board computer system (SBC) (such as, forexample, a computer-on-module (COM) or system-on-module (SOM)), aspecial-purpose computing device, a desktop computer system, a laptop ornotebook computer system, a mobile telephone, a server, a tabletcomputer system, or a combination of two or more of these. Whereappropriate, computer system 2200 may: include one or more computersystems 2200; be unitary or distributed; span multiple locations; spanmultiple machines; span multiple data centers; or reside partly orwholly in a computing cloud, which may include one or more cloudcomputing components in one or more networks. Where appropriate, one ormore computer systems 2200 may perform without substantial spatial ortemporal limitation one or more steps of one or more methods describedor illustrated herein. As an example and not by way of limitation, oneor more computer systems 2200 may perform in real time or in batch modeone or more steps of one or more methods described or illustratedherein. One or more computer systems 2200 may perform at different timesor at different locations one or more steps of one or more methodsdescribed or illustrated herein, where appropriate.

In particular embodiments, computer system 2200 includes at least oneprocessor 2202, memory 2204, storage 2206, an input/output (I/O)interface 2208, a communication interface 2210, and a bus 2212. Althoughthis disclosure describes and illustrates a particular computer systemhaving a particular number of particular components in a particulararrangement, this disclosure contemplates any suitable computer systemhaving any suitable number of any suitable components in any suitablearrangement.

In particular embodiments, processor 2202 includes hardware forexecuting instructions, such as those making up a computer program. Asan example and not by way of limitation, to execute instructions,processor 2202 may retrieve (or fetch) the instructions from an internalregister, an internal cache, memory 2204, or storage 2206; decode andexecute them; and then write one or more results to an internalregister, an internal cache, memory 2204, or storage 2206. In particularembodiments, processor 2202 may include one or more internal caches fordata, instructions, or addresses. This disclosure contemplates processor2202 including any suitable number of any suitable internal caches,where appropriate. As an example and not by way of limitation, processor2202 may include one or more instruction caches, one or more datacaches, and one or more translation lookaside buffers (TLBs).Instructions in the instruction caches may be copies of instructions inmemory 2204 or storage 2206, and the instruction caches may speed upretrieval of those instructions by processor 2202. Data in the datacaches may be copies of data in memory 2204 or storage 2206 forinstructions executing at processor 2202 to operate on; the results ofprevious instructions executed at processor 2202 for access bysubsequent instructions executing at processor 2202 or for writing tomemory 2204 or storage 2206; or other suitable data. The data caches mayspeed up read or write operations by processor 2202. The TLBs may speedup virtual-address translation for processor 2202. In particularembodiments, processor 2202 may include one or more internal registersfor data, instructions, or addresses. This disclosure contemplatesprocessor 2202 including any suitable number of any suitable internalregisters, where appropriate. Where appropriate, processor 2202 mayinclude one or more arithmetic logic units (ALUs); be a multi-coreprocessor; or include one or more processors 2202. Although thisdisclosure describes and illustrates a particular processor, thisdisclosure contemplates any suitable processor.

In particular embodiments, memory 2204 includes main memory for storinginstructions for processor 2202 to execute or data for processor 2202 tooperate on. As an example and not by way of limitation, computer system2200 may load instructions from storage 2206 or another source (such as,for example, another computer system 2200) to memory 2204. Processor2202 may then load the instructions from memory 2204 to an internalregister or internal cache. To execute the instructions, processor 2202may retrieve the instructions from the internal register or internalcache and decode them. During or after execution of the instructions,processor 2202 may write one or more results (which may be intermediateor final results) to the internal register or internal cache. Processor2202 may then write one or more of those results to memory 2204. Inparticular embodiments, processor 2202 executes only instructions in oneor more internal registers or internal caches or in memory 2204 (asopposed to storage 2206 or elsewhere) and operates only on data in oneor more internal registers or internal caches or in memory 2204 (asopposed to storage 2206 or elsewhere). One or more memory buses (whichmay each include an address bus and a data bus) may couple processor2202 to memory 2204. Bus 2212 may include one or more memory buses, asdescribed below. In particular embodiments, one or more memorymanagement units (MMUs) reside between processor 2202 and memory 2204and facilitate accesses to memory 2204 requested by processor 2202. Inparticular embodiments, memory 2204 includes random access memory (RAM).This RAM may be volatile memory, where appropriate. Where appropriate,this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, whereappropriate, this RAM may be single-ported or multi-ported RAM. Thisdisclosure contemplates any suitable RAM. Memory 2204 may include one ormore memories 2204, where appropriate. Although this disclosuredescribes and illustrates particular memory, this disclosurecontemplates any suitable memory.

In particular embodiments, storage 2206 includes mass storage for dataor instructions. As an example and not by way of limitation, storage2206 may include a hard disk drive (HDD), a floppy disk drive, flashmemory, an optical disc, a magneto-optical disc, magnetic tape, or aUniversal Serial Bus (USB) drive or a combination of two or more ofthese. Storage 2206 may include removable or non-removable (or fixed)media, where appropriate. Storage 2206 may be internal or external tocomputer system 2200, where appropriate. In particular embodiments,storage 2206 is non-volatile, solid-state memory. In particularembodiments, storage 2206 includes read-only memory (ROM). Whereappropriate, this ROM may be mask-programmed ROM, programmable ROM(PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM),electrically alterable ROM (EAROM), or flash memory or a combination oftwo or more of these. This disclosure contemplates mass storage 2206taking any suitable physical form. Storage 2206 may include one or morestorage control units facilitating communication between processor 2202and storage 2206, where appropriate. Where appropriate, storage 2206 mayinclude one or more storages 2206. Although this disclosure describesand illustrates particular storage, this disclosure contemplates anysuitable storage.

In particular embodiments, I/O interface 2208 includes hardware,software, or both, providing one or more interfaces for communicationbetween computer system 2200 and one or more I/O devices. Computersystem 2200 may include one or more of these I/O devices, whereappropriate. One or more of these I/O devices may enable communicationbetween a person and computer system 2200. As an example and not by wayof limitation, an I/O device may include a keyboard, keypad, microphone,monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet,touch screen, trackball, video camera, another suitable I/O device or acombination of two or more of these. An I/O device may include one ormore sensors. This disclosure contemplates any suitable I/O devices andany suitable I/O interfaces 2208 for them. Where appropriate, I/Ointerface 2208 may include one or more device or software driversenabling processor 2202 to drive one or more of these I/O devices. I/Ointerface 2208 may include one or more I/O interfaces 2208, whereappropriate. Although this disclosure describes and illustrates aparticular I/O interface, this disclosure contemplates any suitable I/Ointerface.

In particular embodiments, communication interface 2210 includeshardware, software, or both providing one or more interfaces forcommunication (such as, for example, packet-based communication) betweencomputer system 2200 and one or more other computer systems 2200 or oneor more networks. As an example and not by way of limitation,communication interface 2210 may include a network interface controller(NIC) or network adapter for communicating with a wireless adapter forcommunicating with a wireless network, such as a WI-FI or a cellularnetwork. This disclosure contemplates any suitable network and anysuitable communication interface 2210 for it. As an example and not byway of limitation, computer system 2200 may communicate with an ad hocnetwork, a personal area network (PAN), a local area network (LAN), awide area network (WAN), a metropolitan area network (MAN), or one ormore portions of the Internet or a combination of two or more of these.One or more portions of one or more of these networks may be wired orwireless. As an example, computer system 2200 may communicate with awireless cellular telephone network (such as, for example, a GlobalSystem for Mobile Communications (GSM) network, or a 3G, 4G or 5Gcellular network), or other suitable wireless network or a combinationof two or more of these. Computer system 2200 may include any suitablecommunication interface 2210 for any of these networks, whereappropriate. Communication interface 2210 may include one or morecommunication interfaces 2210, where appropriate. Although thisdisclosure describes and illustrates a particular communicationinterface, this disclosure contemplates any suitable communicationinterface.

In particular embodiments, bus 2212 includes hardware, software, or bothcoupling components of computer system 2200 to each other. As an exampleand not by way of limitation, bus 2212 may include an AcceleratedGraphics Port (AGP) or other graphics bus, an Enhanced Industry StandardArchitecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT)interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBANDinterconnect, a low-pin-count (LPC) bus, a memory bus, a Micro ChannelArchitecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, aPCI-Express (PCIe) bus, a serial advanced technology attachment (SATA)bus, a Video Electronics Standards Association local (VLB) bus, oranother suitable bus or a combination of two or more of these. Bus 2212may include one or more buses 2212, where appropriate. Although thisdisclosure describes and illustrates a particular bus, this disclosurecontemplates any suitable bus or interconnect.

Herein, a computer-readable non-transitory storage medium or media mayinclude one or more semiconductor-based or other integrated circuits(ICs) (such, as for example, field-programmable gate arrays (FPGAs) orapplication-specific ICs (ASICs)), hard disk drives (HDDs), hybrid harddrives (HHDs), optical discs, optical disc drives (ODDs),magneto-optical discs, magneto-optical drives, (FDDs), solid-statedrives (SSDs), RAM-drives, or any other suitable computer-readablenon-transitory storage media, or any suitable combination of two or moreof these, where appropriate. A computer-readable non-transitory storagemedium may be volatile, non-volatile, or a combination of volatile andnon-volatile, where appropriate.

The scope of this disclosure encompasses all changes, substitutions,variations, alterations, and modifications to the example embodimentsdescribed or illustrated herein that a person having ordinary skill inthe art would comprehend. The scope of this disclosure is not limited tothe example embodiments described or illustrated herein. Moreover,although this disclosure describes and illustrates respectiveembodiments herein as including particular components, elements,feature, functions, operations, or steps, any of these embodiments mayinclude any combination or permutation of any of the components,elements, features, functions, operations, or steps described orillustrated anywhere herein that a person having ordinary skill in theart would comprehend. Furthermore, reference in the appended claims toan apparatus or system or a component of an apparatus or system beingadapted to, arranged to, capable of, configured to, enabled to, operableto, or operative to perform a particular function encompasses thatapparatus, system, component, whether or not it or that particularfunction is activated, turned on, or unlocked, as long as thatapparatus, system, or component is so adapted, arranged, capable,configured, enabled, operable, or operative. Additionally, although thisdisclosure describes or illustrates particular embodiments as providingparticular advantages, particular embodiments may provide none, some, orall of these advantages.

It will be appreciated by persons skilled in the art that numerousvariations and/or modifications may be made to the above-describedembodiments, without departing from the broad general scope of thepresent disclosure. The present embodiments are, therefore, to beconsidered in all respects as illustrative and not restrictive.

1. A system for location determination, the system comprising: acomputing device comprising at least one processor and a memoryaccessible to the at least one processor; wherein the memory comprises alibrary of reference background images and metadata for each referencebackground image, wherein the metadata comprises location information;and wherein the memory stores program code executable by the at leastone processor to configure the at least one processor to: receive aninput image data from a remote computing device, wherein the input imagedata includes image data of at least one image captured by the remotecomputing device at a location to be determined; process the receivedinput image data using a background matching module to identify matchingreference background image; determine location information correspondingto the input image data based on the metadata of the matching referencebackground image in the library; and transmit the determined locationinformation to the remote computing device.
 2. The system of claim 1,wherein the background matching module comprises: a background featureextractor neural network, and the at least one processor is furtherconfigured to identify the matching reference background image by:extracting background descriptors from the at least one captured imageusing the background feature extractor neural network; selecting one ormore candidate matching images from the library of background imagesbased on the extracted background descriptors; performing geometricmatching between the at least one captured image and the candidatematching images to select the matching reference background image. 3.The system of claim 2, wherein the geometric matching comprisesidentifying common visual features in the at least one captured imageand each of the candidate matching images.
 4. The system of claim 2,wherein the geometric matching is performed using a random sampleconsensus process.
 5. The system of claim 2, wherein the backgroundfeature extractor neural network is trained to extract backgrounddescriptors corresponding to one or more stationary features in the atleast one captured image.
 6. The system of 1, wherein the memory storesprogram code executable by the at least one processor to furtherconfigure the at least one processor to: receive GPS data correspondingto the input image from the remote computing device, wherein the GPSdata comprises a low data quality indicator; information; generate a GPScorrection signal based on the determined location transmit the GPScorrection signal to the remote computing device.
 7. A vehicle mountedsystem for location determination in an urban area, the systemcomprising: at least one camera, wherein the at least one camera ispositioned to capture images of the urban area; a computing device incommunication with the at least one camera to receive the capturedimages, the computing device comprising at least one processor and amemory accessible to the at least one processor, the memory comprising alibrary of reference background images; wherein the memory comprisesprogram code executable by the at least one processor to configure theat least one processor to: extract background descriptors from at leastone captured image; select one or more candidate matching referenceimages from the library of background images based on the extractedbackground descriptors; perform geometric matching between the at leastone captured image and the one or more candidate matching referenceimages to select a single matching reference background image; anddetermine a location of the vehicle based on the single matchingreference background image.
 8. The system of claim 7, wherein the memoryfurther comprises location metadata corresponding to each referencebackground image; and the location of the vehicle is determined based onthe location metadata corresponding to corresponding to the singlematching reference background image.
 9. The system of claim 7, whereinthe background descriptors are extracted from at least one capturedimage using a background feature extractor neural network.
 10. Thesystem of claim 7, wherein determining the location is performed in realtime.
 11. A computer implemented method for determining a location of avehicle, the method performed by a vehicle mounted computing devicecomprising at least one processor in communication with a memory, thememory comprising a library of reference background images, the methodcomprising: receiving images of an urban area captured by a camera incommunication with the computing device; extracting backgrounddescriptors from at least one image captured by the camera; selectingone or more candidate matching reference images from the library ofbackground images based on the extracted background descriptors; performgeometric matching between the at least one captured image and the oneor more candidate matching reference images to select a single matchingreference background image; and determine a location of the vehiclebased on the single matching reference background image.
 12. A computerimplemented method for determining a location of a vehicle in an urbanarea, the method performed by a vehicle mounted computing devicecomprising at least one processor in communication with a memory and atleast one camera, the memory comprising a library of referencebackground images, the method comprising: capturing an image of theurban area while the at least one camera is moving in the urban area;processing the captured image using a background matching module toidentify a matching reference background image; determining a locationof the vehicle based on a metadata of the matching reference backgroundimage.
 13. The method of claim 12, wherein the at least one camera ismounted on the vehicle.
 14. The method of claim 12, wherein thedetermination of the location of the vehicle is performed in real timeby the vehicle mounted computing device.
 15. A system for locationdetermination in an urban area, the system comprising: at least onecamera, wherein the at least one camera is positioned to capture imagesof the urban area while the at least one camera is moving in the urbanarea; a computing device moving with the at least one camera and incommunication with the at least one camera to receive the capturedimages; the computing device comprising at least one processor and amemory accessible to the at least one processor; wherein the memorycomprises a library of reference background images and metadata for eachreference background image, wherein the metadata comprises locationinformation; wherein the memory stores program code executable by the atleast one processor to configure the at least one processor to: processa captured image using a background matching module to identify amatching reference background image; determine a location of the atleast one camera and the computing device based on the metadata of thematching reference background image.
 16. The system of claim 15, whereinprocessing the captured image using a background matching modulecomprises: extracting background descriptors from the captured image;selecting one or more candidate matching images from the library ofreference background images based on the extracted backgrounddescriptors; performing geometric matching between the captured imageand the candidate matching images to select the matching referencebackground image.
 17. The system of claim 16, wherein the backgroundmatching module comprises a background feature extractor neural networkconfigured to extract background descriptors corresponding to one ormore stationary features in the at least one captured image.
 18. Thesystem of claim 16, wherein the geometric matching is performed using arandom sample consensus process; and wherein the geometric matchingcomprises identifying common visual features in the at least onecaptured image and each of the candidate matching images.
 19. The systemof claim 15, wherein the computing device is configured to determine thelocation in real-time.
 20. A vehicle mounted with the system of claim15, wherein the at least one camera is mounted on the vehicle to captureimages of a vicinity of the vehicle.